DOCOHENT BESOHE 



ED 142 552 SP Oil 381 



TITLE Evaluation Plan Model, 

mSTITOTION Texas Education Agency, Austin, 
PDB DATE May 7U 

NOTE 16p, 

EDES PRICE MF-$0.83 HC-$1.67 Plus Postage, 

DESCEIPTORS *Educational Assessment; Educational Programs; 

Evaluation Criteria; Evaluation Methods; ♦Mo'cLels; 

♦Program Evaluation 

ABSTRACT 

A model for the evaluation of educational programs is 
presented, consisting of a seven*- step procedure for determining 
objectives, developing the assessment data, analyzing the data, and 
presenting it in a meaningful format. The first step is to determine 
the objectives for the program and enumerate them in a meaningful, 
measurable fashion; second, to identify criteria bearing upon and 
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purpose of the evaluation; fourth, to develop methods of data 
collection and instrument administration; fifth, to collect data and 
direct it to the proper sources for analysis; sixth, to analyze the 
data, and seventh, to arrange the results in a meaningful report 
format and to arrange distribution. (MJB) 
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ERIC 



COMPLIANCE WITH TITLE VI. CIVIL RIGHTS ACT OF 
1964 AND THE MODIFIED COURT ORDER. CIVIL ACTION 
5281, FEDERAL DISTRICT COURT. EASTERN DISTRICT 
OF TEXAS, TYLER DIVISION 

Reviews of local education agencits periaming to comp'.iance with Title 
VI Civil Rights Act of 1964 and with specific requirements of the 
Modified Coun Order, Civil Action No 5281. Federal District Court, 
Eastern District of Texas. Tyler Division a-e conducted periodically by 
staff representatives of the Texas Educaiion Agency. These reviews 
cover at least the following policies and practices. 

(1) acceptance po' "ics on itudcni transfers from other school 
districts: 

(21 operation of school b is oute or runs on a non-segregated 
basis; 

(31 non discriminjlion in extracurricular activities jnd the use 
of school facilities; 

(4> non discriminatory practices in the hiring, assigning, pro- 
moting, paying, demo^inn. reassigning or dismissing of faculty 
and staff members who work with children; 

(5) enrollment and assignment of students without discrimina- 
tion on the ground of race, color or national oriqin; and 

(6> evidence of published procoilures for hearing complaints and 
grievances. 

In addition to conducting reviews, the Texas Education Agency slijff 
representatives check complaints of discrimination made by a citizen or 
citizens residing in a school district where * is alleged discriminatory 
practices have or are occurring. 

Where a violation of Title of the Civil Rights Act is foii- i. the 
findings are reported to the Office for Civil Riyh... Depar; int of 
Health. Education and Welfare- 

If there be a direct violation of the Co ,rt Order in Civ' Action No. 
5281 that cannot be cleared through negotiation, the sanctions ,r-i'nc(i 
by the Court Order are applied. 
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FOREWORD 



This booklet is being issued in response to requests from 
local, regional, and state education personnel for assis- 
t'^nce in planning and conducting evaluations. Its purpose 
is to provide managers and staff alike with a clear, concise 
outline that will assist in the development of an evaluation 
plan. 

The evaluation effort will depend on the size, location, and 
individual makeup of each educational organization. However, 
the demand for accountability and the need for planned eval- 
uations are common to all. 

We hope this J>ooklet will help concerned educators meet their 
increasing obligations in evaluation. 



J, W, Edgar 

Commissioner of Education 
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INTRODUCTION 



In order to conduct an evaluation which will servo a useful 
purpose for decision ma'king, a plan must be established. The 
plan will provide a means by which the evaluation can be con- 
ducted step-by-step through a predetermined process to a valu- 
able conclusion. The antithesis of this procedure is haphaz- 
ardly groping through a maze of channels gathering liberal 
amounts of data which may or may not relate to one another and 
which ultimately may or may not assist in formulating proper 
decisions. The plan will also provide others a view of the 
evaluation process and will enable those unfamiliar with the 
endeavor to assume responsibility with minimum difficulty should 
a contingency arise. The information rendered by a planned eva 1 
uation can enable program managers to maT<e enlightened decisions 
about those activities under review.^ The following model em- 
braces seven specific steps wTiich' \VTll guide in constructing 
an evaluation plan. (See Appendix A) In order to establish 
a basis of accord regarding the terminology \ised in the text 
of the "Evaluation Plan-Model," each term that is unique to eval 
uation activities is defined in Appendix C. 




DETERMINE 
OBJECTIVES 



The first step in formulating an evaluation plan is to deter- 
mine the objectives of the program (or project) that is to be 
evaluated. One prime aim in this endeavor is to ='ssure that 
each objective is meaningful to all who read it and each reader 
understands clearly what is to be accomplished to complete the 
objective. The second prime aim is to make sure that the reader 
can visualize a practical and effective way or ways to measure 
the thing that is to be accomplished, or the product, and deter- 
mine whether the objective has been met. If the objectives 
are so general that the ultimate product is not clear then it 
cannot be determined that anything of substance has been pro- 
duced. For this reason each objective must communicate clearly 
what is to be accomplished and how accomplishment may be mea- 
sured. When objectives are both meaningful and measurable, the 
evaluator should list those which will be addressed in the eva lua 
tion. This may include all or a select group of the program 
objectives . 



Step two of the plan must mal<e provision for those criteria 
which will influence Mie operation of: the urogram. Include 
here any standards, guidelines* or directives related to the 
operation of the program. When identifying and solectinn 
these criteritj, the ev.-=!' lua tor will find a review of documents 
and correspondence pertinent to the establishment of the pro- 
gram helpful. The evaliator shf^uld ensure that the selected 
criteria are applicable and manageable. At the conclusion of 
this starch the evaxuator should list those criteria determined 
to be appropriate. This list may be altered at a later date 
if circumstances dictate. 




IDENTIFY 
CRITERIA 
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DETERMINE 
PURPOSE 
AND EXTENT 

The third step is to determine the extent and purpose of the 
evaluation to aid in making decisions about the program. This 
will dictate the type of evaluation to be conducted (such as 
process or termina I or both), the kind of data to be collected, 
and the data sources. 

Data may be used primarily to conduct a "terminal 
e-aluation" at the conclusion of the program. This 
would be essentially progress data which is utilized 
to reveal the extent to which program objectives have 
been attained, ilowever, if time and circumstances 
permit the collection of portions of the data for 
"process evaluation," the program manager may have an 
opportunity to detect and correct weaknesses while 
the program is in opera t ion , which wou Id increase 
the value of the overa 11, eva lua tion , Process eva 1- 
uations are ongoing, that is to say they are accom- 
plished at predetermined intervals while the pro- 
gran is operational. It may be appropria*-e to refer 
to them as "mini" evaluations since they address 
only segments of the whole, while a terminal evalua- 
tion addresses the total program after it is con- 
cluded. Process evaluation may be included in an 
evaluation plan as frequently as resources, circum- 
stances, an(3 the length and complexity of the pro-' 
g^am pe rmit . Howeve r , an eva lua t ion pla n shou Id 
include only one terminal evaluation. 

The kind of data to be collected may be either fjct 
or opinion or both. This will be governed by the 
decisions to be made. In order to accumulate the 
right data upon which to base decisions, the evalua- 
tor should establish a set of prime questions on a 
variety of pertinent subjects. The questions should 
be structured with direct relationship to the pro- 
gram objectives. Then for each prime question the 
evaluator must decide what will be an acceptable 




4 



answer to thrjt question. The evaluator should deter- 
mine what data will provide the answer. Data accu- 
mulated through survey and testing programs must be 
adapatable to manual or computer manipulation. 

Equally important is det^'jrmining the sources from 
which the data will be obtained. Careful attention 
should be directed to selecting the sources. There 
are usually many sowTces for collecting data, but 
first consideration should be given to the segments 
of the population or organization that have some 
degree of association or relationship with the pro- 
gram, Th\s does- not imply that people are the only 
sources of data, because often much can be gleaned 
from records, reports , test scorts , and the like • 

For this portion of the plan the evaluator should record the 
'type of evaluation to be conducted along with starting and 
ending dates. The evaluator should list each prime question, 
that which will be an acceptable answer, thnt data needed to 
provide the answer, and the data sources. 

The following is an example of the four elements mentioned 
above (this example is one of numerous prime questions used 
in an evaluation plan for a program designed to reorganize 
the operating procedures of a school district): 

Question - Has the curriculum demonstrated flexi- 
bility and .continuity of courses during each of 
the attendance periods of the school year? 

Acceptable A \r - If time was provided for the 
teachers to rc ganize the instructiona 1 pattern, 
the scheduling of courses was accomplished easily, 
and students were easily scheduled, the curriculum 
"had flexibility and continuity. 

Data - Information on time required to reorganize 
the instructional pattern and problems experienced 
in course scheduling. The list of courses offered 
during the current and past school years. 

Sources - School principals, teachers, counselors, 
secondary students, and records. 
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COLLECT 
DATA 



The fourth step includes procedures for developing the dnta 
collection techniques and administr.ntion • The evalucitor should 
include the amount of data to be accumulated, the time periods, 
the frequer:cy, and the instrumentation, 

A review of the prime questions established in step 
three in concert with the type of evnluation to be 
conducted will aid in establishing the time periods 
and frequency of the data collection activities. 

Too much or too little data may handicap the pros- 
pects of a successful evaluation. The evaluator 
should consider each data* collection source when 
deterrnining whether to employ either a universnl 
or sample technique. The extent of the evaluation, 
the time allocated, t;he frequency of activities, 
and the resources yv-<ilable for treatment and anal- 
ysis of the dat*'i 11 determine tho techniques to 
be used. 

A review of the pri ^e questions and data souirces 
established in step three will aid in choosing the 
appropriate da ha collection methods. Tests, ques- 
tionnaires, interviews, meetings, and reviews of 
reports and records are ysed most frequently. The 
use of a variety of instrviments is recommended. 
The instruments should be constructed so that each 
prime question is addressed by at least two separate 
data collection sources. 

The evaluator should record when, how much, and how often to 
collect data and what data collection techniques to use. 
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PROCESS 
DATA 



> 



The proposal for processing the data comprises the fift'i step. 

The evaluator should request that a contact person 
be identified by each major activity from which data 
• is to be collected. This person should be provided 
sufficient in forma t ion and instructions to ensure 
that the data are collected and disseminated to the 
proper sources for treatment. 

When data are to be collected from records or reports 
the evaluator shou 3 J arrange for that data to be 
compiled by the activity responsible for the source 
documents. 

When questionnaires or similar survey instiruments 
are used to collect data, the evaluator should be 
the prime agent for distributing and collecting the 
instniments. Arrangements rhould be made to ensure 
that the instruments are handled by the minimum ' 
number of people. 

The evaluator should ensure that interviews are con- 
ducted by professional interviewers or that people 
are selected for their ability and are provided inter- 
view training. 

When pre- and post-test data are used the evaluator 
should <?nsure their conf identia lity. 

The data processing procedures and responsibilities are go be 
outlined in this part of the plan. The suggestions above pro- 
vide some guides for structuring this outline. It is generally 
better to identify responsibilities with positions and functions 
rather than with personalities. 
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ANALYZE 



DATA 



The sixth step is t)ie procedure for emalyzing the data- This 
is the eva lua tion . In order to determine the effectiveness of 
the program, the accumulated data should be compared through 
the use of one or more of the following systems: 

The evaluator may establish criteria upon which to 
judge the 'effectiveness of the program in meeting 
program objectives. The data collected ma y i:hon be 
compared against the established criteria. 

Specific program data may be compared Lo the opera- 
tional data accumulated f rom'pre^iotis years. 

Specific program duta may be compared to similiir 
data accumulated from a program conducted for the 
same purpose. (The programs for comparison should 
be as nearly equivalent in al] renditions ^is is 
possible . ) 

The evaluator should also determine D-.o ox ton t of adherence to 
the criteria in the iniplementation of the ovviluation plan. This 
information will be useful in interpret i nq the findings. The 
evaluator should describe the procedures for annlyzing the accu- 
mulated data. The prime questions ost.ibi i .shod in i^tep tliree 
are the key to this phase of tlie eva 1 1:.^ t i on . 



The seventh step is to establish t:l e rofrndl jntl toc>inic|nes for 
reporting the evaluation rostiRs, Thi- c-vn lncition report stioiild 
then oe produced and distribU'-od t]r:.\so whrj ova to mn\:e tlie 
decisions about the program. r'.vri ja t. i un ^hotild he <*i continninq 
process witli findings .^.--nt to tho U'c: is ■. o'.-i rv'iV.erii ::-n ,\ rogular 
basis . 




REPORT 
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APPENDIX B 



MODEL SUMMARIZED 

This model has been developed to aid in devising an evaluation 
plan to determining the effectiveness of a program. 

There are seven steps involved: 

The first s**ep is to determine the objectives for 
the program. The objectives should be meaningful 
and measurable. 

The second step is tc identify the criteria which 
will bear upon and influence the program operation. 
The criteria should be applicable and manageable. 

The third step is to determine the extent and pur- 
pose of the evaluation. This decision will deter- 
mine the type of evaluation, the kind of data, and 
t^e data sources. 

The fourth step involves developing the methods of 
collecting data and the c^dministra tion of the data 
collection procedures . During th is step it is deter- 
mined when, how much, and how often to collect data 
arid what dato collection techniques to use. 

The fifth step is collecting and directing the data 
to the proper sources for treatment. !Iere the res- 
ponsibilities for collecting and processing the 
data to the users are identified. 

The sixth step is analyzing the data. This is the 
treatment of the data to determine the actual effec- 
tiveness of the program. This activity should re- 
veal to what degree each objective was acccomplished 
and provide guidance for future decisions. 

The seventh and last step is arranging the results 
in a meaningful report format ard distributing this 
to the people who will be making decisions about this 
program. 
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APPENDIX C 



GLOSSARY OF TERMS 

Meaningful objective - a statement that describes the thing one 
intends to accomplish without being vague or misleading, 

Measureable objective - a statement that describes an intended 
accomplishment which is capable of being measured effec- 
tively. 

Process evaluation - an evaluation that is conducted to judge 
how the program is progressing tcward production of the 
intended product. Data are collected during a partial 
period of t'ne program; after analysis the conclusions 
should indicate whether changes or adjustments should be 
made in the program or that the program is progressing 
satisiractorily. 

"Terminal evaluation - an evaluation that is conducted to judge 
the final product when the program is completed. Data for 
this purpose may be collected while the program is being 
conducted, but the final analysis is ^.not .par formed nor the 
conclujaions formulated until after th(?;,;program is completed 

. - "* 

Criterion - a standard that a judgment may be based upon. 

Applicable criterion - a standard that is compatible with the 
circumstances of the program and can bo applied to make a 
judgment. 

Manageable criterion - a standard that is manageable in accor- 
dance with the resources available anj the time allocated 
to effecU a judgment. 
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